通过 docker |
您所在的位置：网站首页 › docker-compose mysql连接不上 › 通过 docker |
通过 docker
|
一、概述
其实通过 docker-compose 部署 hive 是在继上篇文章 Hadoop 部署的基础之上叠加的，Hive 做为最常用的数仓服务，所以是有必要进行集成的，感兴趣的小伙伴请认真阅读我以下内容，通过 docker-compose 部署的服务主要是用最少的资源和时间成本快速部署服务，方便小伙伴学习、测试、验证功能等等~ 关于 Hadoop 部署可以查阅我以下几篇文章： 通过 docker-compose 快速部署 Hadoop 集群详细教程 通过 docker-compose 快速部署 Hadoop 集群极简教程最好是先浏览一下Hadoop 部署的文章，如果不 care 详细过程，就可以只查阅 通过 docker-compose 快速部署 Hadoop 集群极简教程 这篇文章即可~ 关于 Hive 的介绍可以查阅我以下文章：大数据Hadoop之——数据仓库Hive 二、前期准备 1）部署 docker # 安装yum-config-manager配置工具 yum -y install yum-utils # 建议使用阿里云yum源：（推荐） #yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 安装docker-ce版本 yum install -y docker-ce # 启动并开机启动 systemctl enable --now docker docker --version 复制代码 2）部署 docker-compose curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-compose --version 复制代码 三、创建网络 # 创建，注意不能使用hadoop-network，要不然启动hs2服务的时候会有问题！！！ docker network create hadoop-network # 查看 docker network ls 复制代码 四、MySQL 部署 1）mysql 镜像 docker pull mysql:5.7 docker tag mysql:5.7 registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql:5.7 复制代码 2）配置 mkdir -p conf/ data/db/ cat >conf/my.cnf javax.jdo.option.ConnectionURL jdbc:mysql://mysql:3306/hive_metastore?createDatabaseIfNotExist=true;useSSL=false;serverTimezone=Asia/Shanghai javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver javax.jdo.option.ConnectionUserName root javax.jdo.option.ConnectionPassword 123456 hive.metastore.schema.verification false system:user.name root user name hive.metastore.uris thrift://hive-metastore:9083 hive.server2.thrift.bind.host 0.0.0.0 Bind host on which to run the HiveServer2 Thrift service. hive.server2.thrift.port 10000 hive.server2.active.passive.ha.enable true 复制代码 3）启动脚本 #!/usr/bin/env sh wait_for() { echo Waiting for $1 to listen on $2... while ! nc -z $1 $2; do echo waiting...; sleep 1s; done } start_hdfs_namenode() { if [ ! -f /tmp/namenode-formated ];then ${HADOOP_HOME}/bin/hdfs namenode -format >/tmp/namenode-formated fi ${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start namenode tail -f ${HADOOP_HOME}/logs/*namenode*.log } start_hdfs_datanode() { wait_for $1 $2 ${HADOOP_HOME}/bin/hdfs --loglevel INFO --daemon start datanode tail -f ${HADOOP_HOME}/logs/*datanode*.log } start_yarn_resourcemanager() { ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start resourcemanager tail -f ${HADOOP_HOME}/logs/*resourcemanager*.log } start_yarn_nodemanager() { wait_for $1 $2 ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start nodemanager tail -f ${HADOOP_HOME}/logs/*nodemanager*.log } start_yarn_proxyserver() { wait_for $1 $2 ${HADOOP_HOME}/bin/yarn --loglevel INFO --daemon start proxyserver tail -f ${HADOOP_HOME}/logs/*proxyserver*.log } start_mr_historyserver() { wait_for $1 $2 ${HADOOP_HOME}/bin/mapred --loglevel INFO --daemon start historyserver tail -f ${HADOOP_HOME}/logs/*historyserver*.log } start_hive_metastore() { if [ ! -f ${HIVE_HOME}/formated ];then schematool -initSchema -dbType mysql --verbose > ${HIVE_HOME}/formated fi $HIVE_HOME/bin/hive --service metastore } start_hive_hiveserver2() { $HIVE_HOME/bin/hive --service hiveserver2 } case $1 in hadoop-hdfs-nn) start_hdfs_namenode ;; hadoop-hdfs-dn) start_hdfs_datanode $2 $3 ;; hadoop-yarn-rm) start_yarn_resourcemanager ;; hadoop-yarn-nm) start_yarn_nodemanager $2 $3 ;; hadoop-yarn-proxyserver) start_yarn_proxyserver $2 $3 ;; hadoop-mr-historyserver) start_mr_historyserver $2 $3 ;; hive-metastore) start_hive_metastore $2 $3 ;; hive-hiveserver2) start_hive_hiveserver2 $2 $3 ;; *) echo "请输入正确的服务启动命令~" ;; esac 复制代码 4）构建镜像 Dockerfile FROM registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop:v1 COPY hive-config/* ${HIVE_HOME}/conf/ COPY bootstrap.sh /opt/apache/ COPY mysql-connector-java-5.1.49/mysql-connector-java-5.1.49-bin.jar ${HIVE_HOME}/lib/ RUN sudo mkdir -p /home/hadoop/ && sudo chown -R hadoop:hadoop /home/hadoop/ #RUN yum -y install which 复制代码开始构建镜像 # 构建镜像 docker build -t registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 . --no-cache # 推送镜像（可选） docker push registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 ### 参数解释 # -t：指定镜像名称 # . ：当前目录Dockerfile # -f：指定Dockerfile路径 # --no-cache：不缓存 复制代码 5）编排 version: '3' services: hadoop-hdfs-nn: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-hdfs-nn hostname: hadoop-hdfs-nn restart: always privileged: true env_file: - .env ports: - "30070:${HADOOP_HDFS_NN_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-nn"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_HDFS_NN_PORT} || exit 1"] interval: 20s timeout: 20s retries: 3 hadoop-hdfs-dn-0: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-hdfs-dn-0 hostname: hadoop-hdfs-dn-0 restart: always depends_on: - hadoop-hdfs-nn env_file: - .env ports: - "30864:${HADOOP_HDFS_DN_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-dn hadoop-hdfs-nn ${HADOOP_HDFS_NN_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_HDFS_DN_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-hdfs-dn-1: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-hdfs-dn-1 hostname: hadoop-hdfs-dn-1 restart: always depends_on: - hadoop-hdfs-nn env_file: - .env ports: - "30865:${HADOOP_HDFS_DN_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-dn hadoop-hdfs-nn ${HADOOP_HDFS_NN_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_HDFS_DN_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-hdfs-dn-2: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-hdfs-dn-2 hostname: hadoop-hdfs-dn-2 restart: always depends_on: - hadoop-hdfs-nn env_file: - .env ports: - "30866:${HADOOP_HDFS_DN_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-hdfs-dn hadoop-hdfs-nn ${HADOOP_HDFS_NN_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_HDFS_DN_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-yarn-rm: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-yarn-rm hostname: hadoop-yarn-rm restart: always env_file: - .env ports: - "30888:${HADOOP_YARN_RM_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-rm"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${HADOOP_YARN_RM_PORT} || exit 1"] interval: 20s timeout: 20s retries: 3 hadoop-yarn-nm-0: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-yarn-nm-0 hostname: hadoop-yarn-nm-0 restart: always depends_on: - hadoop-yarn-rm env_file: - .env ports: - "30042:${HADOOP_YARN_NM_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-nm hadoop-yarn-rm ${HADOOP_YARN_RM_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_YARN_NM_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-yarn-nm-1: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-yarn-nm-1 hostname: hadoop-yarn-nm-1 restart: always depends_on: - hadoop-yarn-rm env_file: - .env ports: - "30043:${HADOOP_YARN_NM_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-nm hadoop-yarn-rm ${HADOOP_YARN_RM_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_YARN_NM_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-yarn-nm-2: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-yarn-nm-2 hostname: hadoop-yarn-nm-2 restart: always depends_on: - hadoop-yarn-rm env_file: - .env ports: - "30044:${HADOOP_YARN_NM_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-nm hadoop-yarn-rm ${HADOOP_YARN_RM_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "curl --fail http://localhost:${HADOOP_YARN_NM_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-yarn-proxyserver: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-yarn-proxyserver hostname: hadoop-yarn-proxyserver restart: always depends_on: - hadoop-yarn-rm env_file: - .env ports: - "30911:${HADOOP_YARN_PROXYSERVER_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-yarn-proxyserver hadoop-yarn-rm ${HADOOP_YARN_RM_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${HADOOP_YARN_PROXYSERVER_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hadoop-mr-historyserver: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hadoop-mr-historyserver hostname: hadoop-mr-historyserver restart: always depends_on: - hadoop-yarn-rm env_file: - .env ports: - "31988:${HADOOP_MR_HISTORYSERVER_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hadoop-mr-historyserver hadoop-yarn-rm ${HADOOP_YARN_RM_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${HADOOP_MR_HISTORYSERVER_PORT} || exit 1"] interval: 30s timeout: 30s retries: 3 hive-metastore: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hive-metastore hostname: hive-metastore restart: always depends_on: - hadoop-hdfs-dn-2 env_file: - .env ports: - "30983:${HIVE_METASTORE_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hive-metastore hadoop-hdfs-dn-2 ${HADOOP_HDFS_DN_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${HIVE_METASTORE_PORT} || exit 1"] interval: 30s timeout: 30s retries: 5 hive-hiveserver2: image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/hadoop_hive:v1 user: "hadoop:hadoop" container_name: hive-hiveserver2 hostname: hive-hiveserver2 restart: always depends_on: - hive-metastore env_file: - .env ports: - "31000:${HIVE_HIVESERVER2_PORT}" command: ["sh","-c","/opt/apache/bootstrap.sh hive-hiveserver2 hive-metastore ${HIVE_METASTORE_PORT}"] networks: - hadoop-network healthcheck: test: ["CMD-SHELL", "netstat -tnlp|grep :${HIVE_HIVESERVER2_PORT} || exit 1"] interval: 30s timeout: 30s retries: 5 # 连接外部网络 networks: hadoop-network: external: true 复制代码 6）开始部署 docker-compose -f docker-compose.yaml up -d # 查看 docker-compose -f docker-compose.yaml ps 复制代码
【问题】如果出现以下类似的错误，是因为多次启动，之前的数据还在，但是datanode的IP是已经变了的（宿主机部署就不会有这样的问题，因为宿主机的IP是固定的），所以需要刷新节点，当然也可清理之前的旧数据，不推荐清理旧数据，推荐使用刷新节点的方式（如果有对外挂载的情况下，像我这里没有对外挂载，是因为之前旧容器还在，下面有几种解决方案）： org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode because the host is not in the include-list: DatanodeRegistration(172.30.0.12:9866, datanodeUuid=f8188476-4a88-4cd6-836f-769d510929e4, infoPort=9864, infoSecurePort=0, ipcPort=9867, storageInfo=lv=-57;cid=CID-f998d368-222c-4a9a-88a5-85497a82dcac;nsid=1840040096;c=1680661390829)
到此，Hive 的容器化部署就完成了，有任何疑问的小伙伴欢迎给我留言，后续会持续更新相关技术文章，也可关注我的公众号【大数据与云原生技术分享】深入交流技术或私信咨询问题~ |
【本文地址】
今日新闻 |
推荐新闻 |